Systems and methods for processing images to prepare slides for processed images for digital pathology

ABSTRACT

Systems and methods are disclosed for processing an electronic image corresponding to a specimen. One method for processing the electronic image includes: receiving a target electronic image of a slide corresponding to a target specimen, the target specimen including a tissue sample from a patient, applying a machine learning system to the target electronic image to determine deficiencies associated with the target specimen, the machine learning system having been generated by processing a plurality of training images to predict stain deficiencies and/or predict a needed recut, the training images including images of human tissue and/or images that are algorithmically generated; and based on the deficiencies associated with the target specimen, determining to automatically order an additional slide to be prepared.

RELATED APPLICATION(S)

This application is a continuation of and claims the benefit of priorityto U.S. application Ser. No. 17/137,769, filed Dec. 30, 2020, which is acontinuation of U.S. application Ser. No. 16/884,978, (now U.S. Pat. No.10,937,541), filed May 27, 2020, which claims priority to U.S.Provisional Application No. 62/853,383, filed May 28, 2019, each ofwhich are incorporated herein by reference in their entireties.

FIELD OF THE DISCLOSURE

Various embodiments of the present disclosure pertain generally topathology slide preparation and related image processing methods. Morespecifically, particular embodiments of the present disclosure relate tosystems and methods for identifying or detecting slides lackinginformation sufficient to provide a diagnosis based on processing imagesof tissue specimens. The present disclosure further provides systems andmethods for automatically ordering additional slides that may containdata sufficient to provide a diagnosis based on processing images oftissue specimens.

BACKGROUND

Pathology specimens may be cut into multiple sections, prepared asslides, and stained for a pathologist to examine and render a diagnosis.When uncertain of a diagnostic finding on a slide, a pathologist mayorder additional cut levels, stains, or other tests to gather moreinformation from the tissue. Technicians may then create new slideswhich may contain additional information for the pathologist to use inmaking a diagnosis. This process of creating additional slides may betime-consuming, not only because it may involve retrieving the block oftissue, cutting it to make a new a slide, and then staining the slide,but also because it may be batched for multiple orders. This maysignificantly delay the final diagnosis that the pathologist renders.Even after the delay, the new slides still may not have informationsufficient to render a diagnosis.

A desire exists for a way to expedite or streamline the slidepreparation process, and to ensure that pathology slides have sufficientinformation to render a diagnosis, by the time the slides are reviewedby a pathologist. Disclosed embodiments ensure that slides may provideinformation sufficient to render a diagnosis, before a pathologistreviews the slide. The disclosed embodiments may save a pathologist fromreviewing slides that provide insufficient information to render adiagnosis.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of thedisclosure. The background description provided herein is for thepurpose of generally presenting the context of the disclosure. Unlessotherwise indicated herein, the materials described in this section arenot prior art to the claims in this application and are not admitted tobe prior art, or suggestions of the prior art, by inclusion in thissection.

SUMMARY

According to certain aspects of the present disclosure, systems andmethods are disclosed for determining to order an additional slide basedon image analysis of tissue specimens from digital pathology images.

A computer-implemented method for processing an electronic imagecorresponding to a specimen includes: receiving a target electronicimage of a slide corresponding to a target specimen, the target specimenincluding a tissue sample from a patient; applying a machine learningsystem to the target electronic image to determine deficienciesassociated with the target specimen, the machine learning system havingbeen generated by processing a plurality of training images to predictstain deficiencies and/or predict a needed recut, the training imagesincluding images of human tissue and/or images that are algorithmicallygenerated; and based on the deficiencies associated with the targetspecimen, determining to automatically order an additional slide to beprepared.

In accordance with another embodiment, a system for processing anelectronic image corresponding to a specimen includes: at least onememory storing instructions; and at least one processor configured toexecute the instructions to perform operations including: receiving atarget electronic image of a slide corresponding to a target specimen,the target specimen including a tissue sample from a patient; applying amachine learning system to the target electronic image to determinedeficiencies associated with the target specimen, the machine learningsystem having been generated by processing a plurality of trainingimages to predict stain deficiencies and/or predict a needed recut, thetraining images including images of human tissue and/or images that arealgorithmically generated; and based on the deficiencies associated withthe target specimen, determining to automatically order an additionalslide to be prepared.

In accordance with another embodiment, a non-transitorycomputer-readable medium storing instructions that, when executed byprocessor, cause the processor to perform a method for processing anelectronic image corresponding to a specimen, the method including:receiving a target electronic image of a slide corresponding to a targetspecimen, the target specimen including a tissue sample from a patient;applying a machine learning system to the target electronic image todetermine deficiencies associated with the target specimen, the machinelearning system having been generated by processing a plurality oftraining images to predict stain deficiencies and/or predict a neededrecut, the training images including images of human tissue and/orimages that are algorithmically generated; and based on the deficienciesassociated with the target specimen, determining to automatically orderan additional slide to be prepared.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the disclosed embodiments, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate various exemplary embodiments andtogether with the description, serve to explain the principles of thedisclosed embodiments.

FIG. 1A is an exemplary block diagram of a system and network fordetermining to order additional slides based on image analysis of tissuespecimens from digital pathology image(s), according to an exemplaryembodiment of the present disclosure.

FIG. 1B is an exemplary block diagram of a disease detection platform100, according to an exemplary embodiment of the present disclosure.

FIG. 1C is an exemplary block diagram of a slide analysis platform 101,according to an exemplary embodiment of the present disclosure.

FIG. 2 is a flowchart illustrating an exemplary method for determiningto order additional slides based on image analysis of tissue specimensfrom digital pathology image(s), using machine learning, according to anexemplary embodiment of the present disclosure.

FIG. 3 is a flowchart of an exemplary method for determining slidepreparation parameters, according to an exemplary embodiment of thepresent disclosure.

FIG. 4 is a flowchart of an exemplary method of generating and using astain order prediction tool, according to an exemplary embodiment of thepresent disclosure.

FIG. 5 is a flowchart of an exemplary method of generating and using arecut order prediction tool, according to an exemplary embodiment of thepresent disclosure.

FIG. 6 depicts an example system that may execute techniques presentedherein.

DESCRIPTION OF THE EMBODIMENTS

Reference will now be made in detail to the exemplary embodiments of thepresent disclosure, examples of which are illustrated in theaccompanying drawings. Wherever possible, the same reference numberswill be used throughout the drawings to refer to the same or like parts.

The systems, devices, and methods disclosed herein are described indetail by way of examples and with reference to the figures. Theexamples discussed herein are examples only and are provided to assistin the explanation of the apparatuses, devices, systems, and methodsdescribed herein. None of the features or components shown in thedrawings or discussed below should be taken as mandatory for anyspecific implementation of any of these devices, systems, or methodsunless specifically designated as mandatory.

Also, for any methods described, regardless of whether the method isdescribed in conjunction with a flow diagram, it should be understoodthat unless otherwise specified or required by context, any explicit orimplicit ordering of steps performed in the execution of a method doesnot imply that those steps must be performed in the order presented butinstead may be performed in a different order or in parallel.

As used herein, the term “exemplary” is used in the sense of “example,”rather than “ideal.” Moreover, the terms “a” and “an” herein do notdenote a limitation of quantity, but rather denote the presence of oneor more of the referenced items.

Pathology refers to the study of diseases. More specifically, pathologyrefers to performing tests and analysis that are used to diagnosediseases. For example, tissue samples may be placed onto slides to beviewed under a microscope by a pathologist (e.g., a physician that is anexpert at analyzing tissue samples to determine whether anyabnormalities exist). That is, pathology specimens may be cut intomultiple sections, prepared as slides, and stained for a pathologist toexamine and render a diagnosis.

Pathologists may evaluate cancer and other disease pathology slides inisolation. The present disclosure presents a consolidated workflow forimproving diagnosis of cancer and other diseases. The workflow mayintegrate, for example, slide evaluation, tasks, image analysis andcancer detection artificial intelligence (AI), annotations,consultations, and recommendations in one workstation. In particular,the present disclosure describes various exemplary user interfacesavailable in the workflow, as well as AI tools that may be integratedinto the workflow to expedite and improve a pathologist's work.

The process of using computers to assist pathologists is known ascomputational pathology. Computing methods used for computationalpathology may include, but are not limited to, statistical analysis,autonomous or machine learning, and AI. AI may include, but is notlimited to, deep learning, neural networks, classifications, clustering,and regression algorithms. By using computational pathology, lives maybe saved by helping pathologists improve their diagnostic accuracy,reliability, efficiency, and accessibility. For example, computationalpathology may be used to assist with detecting slides suspicious forcancer, thereby allowing pathologists to check and confirm their initialassessments before rendering a final diagnosis.

Histopathology refers to the study of a specimen that has been placedonto a slide. For example, a digital pathology image may be comprised ofa digitized image of a microscope slide containing the specimen (e.g., asmear). One method a pathologist may use to analyze an image on a slideis to identify nuclei and classify whether a nucleus is normal (e.g.,benign) or abnormal (e.g., malignant). To assist pathologists inidentifying and classifying nuclei, histological stains may be used tomake cells visible. Many dye-based staining systems have been developed,including periodic acid-Schiff reaction, Masson's trichrome, nissl andmethylene blue, and Haemotoxylin and Eosin (H&E). For medical diagnosis,H&E is a widely used dye-based method, with hematoxylin staining cellnuclei blue, eosin staining cytoplasm and extracellular matrix pink, andother tissue regions taking on variations of these colors. In manycases, however, H&E-stained histologic preparations do not providesufficient information for a pathologist to visually identify biomarkersthat may aid diagnosis or guide treatment. In this situation, techniquessuch as immunohistochemistry (IHC), immunofluorescence, in situhybridization (ISH), or fluorescence in situ hybridization (FISH), maybe used. IHC and immunofluorescence involve, for example, usingantibodies that bind to specific antigens in tissues enabling the visualdetection of cells expressing specific proteins of interest, which mayreveal biomarkers that are not reliably identifiable to trainedpathologists based on the analysis of H&E stained slides. ISH and FISHmay be employed to assess the number of copies of genes or the abundanceof specific RNA molecules, depending on the type of probes employed(e.g. DNA probes for gene copy number and RNA probes for the assessmentof RNA expression). If these methods also fail to provide sufficientinformation to detect some biomarkers, genetic testing of the tissue maybe used to confirm if a biomarker is present (e.g., overexpression of aspecific protein or gene product in a tumor, amplification of a givengene in a cancer).

A digitized image may be prepared to show a stained microscope slide,which may allow a pathologist to manually view the image on a slide andestimate a number of stained abnormal cells in the image. However, thisprocess may be time consuming and may lead to errors in identifyingabnormalities because some abnormalities are difficult to detect.

Computational pathology processes and devices may be used to assistpathologists in detecting abnormalities that may otherwise be difficultto detect. For example, AI may be used to predict biomarkers (such asthe over-expression of a protein and/or gene product, amplification, ormutations of specific genes) from salient regions within digital imagesof tissues stained using H&E and other dye-based methods. The images ofthe tissues could be whole slide images (WSI), images of tissue coreswithin microarrays or selected areas of interest within a tissuesection. Using staining methods like H&E, these biomarkers may bedifficult for humans to visually detect or quantify without the aid ofadditional testing. Using AI to infer these biomarkers from digitalimages of tissues has the potential to improve patient care, while alsobeing faster and less expensive.

The detected biomarkers or the image alone could then be used torecommend specific cancer drugs or drug combination therapies to be usedto treat a patient, and the AI could identify which drugs or drugcombinations are unlikely to be successful by correlating the detectedbiomarkers with a database of treatment options. This may be used tofacilitate the automatic recommendation of immunotherapy drugs to targeta patient's specific cancer. Further, this could be used for enablingpersonalized cancer treatment for specific subsets of patients and/orrarer cancer types.

In the field of pathology today, it may be difficult to providesystematic quality control (“QC”), with respect to pathology specimenpreparation, and quality assurance (“QA”) with respect to the quality ofdiagnoses, throughout the histopathology workflow. Systematic qualityassurance is difficult because it is resource and time intensive as itmay require duplicative efforts by two pathologists. Some methods forquality assurance include (1) second review of first-time diagnosiscancer cases; (2) periodic reviews of discordant or changed diagnoses bya quality assurance committee; and (3) random review of a subset ofcases. These are non-exhaustive, mostly retrospective, and manual. Withan automated and systematic QC and QA mechanism, quality may be ensuredthroughout the workflow for every case. Laboratory quality control anddigital pathology quality control are critical to the successful intake,process, diagnosis, and archive of patient specimens. Manual and sampledapproaches to QC and QA confer substantial benefits. Systematic QC andQA has the potential to provide efficiencies and improve diagnosticquality.

As described above, computational pathology processes and devices of thepresent disclosure may provide an integrated platform allowing a fullyautomated process including data ingestion, processing and viewing ofdigital pathology images via a web-browser or other user interface,while integrating with a laboratory information system (LIS). Further,clinical information may be aggregated using cloud-based data analysisof patient data. The data may come from hospitals, clinics, fieldresearchers, etc., and may be analyzed by machine learning, computervision, natural language processing, and/or statistical algorithms to doreal-time monitoring and forecasting of health patterns at multiplegeographic specificity levels.

As described above, example embodiments described herein determinewhether enough information has been collected from a tissue specimen tomake a diagnosis. For example, computers may be used to analyze an imageof a tissue sample to quickly identify whether additional informationmay be needed about a particular tissue sample, and/or to highlight to apathologist an area in which he or she should look more closely. Whenpaired with automatic slide segmenting and staining machines, this mayprovide a fully automated slide preparation pipeline. This automationhas, at least, the benefits of (1) minimizing an amount of time wastedby a pathologist determining a slide to be insufficient to make adiagnosis, (2) minimizing the (average total) time from specimenacquisition to diagnosis by avoiding the additional time between whenadditional tests are ordered and when they are produced, (3) reducingthe amount of time per recut and the amount of material wasted byallowing recuts to be done while tissue blocks (e.g., pathologyspecimens) are in a cutting desk, (4) reducing the amount of tissuematerial wasted/discarded during slide preparation, (5) reducing thecost of slide preparation by partially or fully automating theprocedure, (6) allowing automatic customized cutting and staining ofslides that would result in more representative/informative slides fromsamples, (7) allowing higher volumes of slides to be generated pertissue block, contributing to more informed/precise diagnoses byreducing the overhead of requesting additional testing for apathologist, and/or (8) identifying or verifying correct properties(e.g., pertaining to a specimen type) of a digital pathology image, etc.

The below embodiments describe various machine learning algorithmtraining methods and implementations. These embodiments are merelyexemplary. Any training methodologies could be used to train a machinelearning model and/or system for the specific purpose of enhancingpathology slide preparation and analysis. Below, some exemplary termsare described.

A whole slide image (WSI) may include an entire scanned pathology slide.A training dataset may include a set of whole slide images and/oradditional diagnostic data from a set of cases used for training themachine learning (ML) algorithm. A validation dataset may include a setof whole slide images and/or additional diagnostic data from a set ofcases used for validating the generalizability of the ML algorithm. Aset of labels may be used for each instance in the training data thatcontain information that an algorithm is being trained to predict (e.g.,whether pathologists requested additional testing/re-cuts for a WSI,etc.). A convolutional neural network (CNN) may refer to an architecturethat may be built that can scan over the WSI. One embodiment may includetraining this CNN, using the training labels, to make one prediction perWSI about the likelihood that additional testing/slide preparation isdesired. A CNN+Aggregator may refer to an architecture that may be builtto incorporate information from a CNN that is executed over multiplelocalized regions of a WSI. One embodiment may include training thisCNN, using the training labels, to make predictions for each region inthe WSI about the likelihood that additional testing/slide preparationmay be needed due to information in a specimen or scanned region. Foradditional levels/cuts, the criteria used may be that staining isinadequate/abnormal, only a small volume of tumor is detected (e.g., forprostate if an atypical small acinar proliferation (ASAP) is detected),if an inadequate amount of tissue is present, tissue folds, etc. Forrescanning, this may include the presence of bubbles, blur, and/orscanning artifacts, etc. More complex training methodologies, such asMultiple Instance Learning, may be used to overcome issues presentedwhen labels do not match one-to-one with WSI regions. In someembodiments, a second model may take individual predictions overtissue/specimen/image regions as inputs and predict the likelihood thatthe WSI may need additional testing/slide preparation. Model Uncertaintymay refer to a machine learning model that may be trained to predict anyparameter about, or related to, a WSI, e.g., detection of a presence ofcancer or other diseases. The level of uncertainty the machine learningmodel has about specific predictions could be computed using a varietyof methods, e.g., identifying an ambiguous range of the probabilityvalues such as those close to the threshold, using out-of-distributiontechniques (Out-of-Distribution detector for Neural Networks (ODIN),tempered mix-up, Mahalanobis distance on the embedding space), etc. Thisuncertainty could be used to estimate the likelihood a slide may needadditional testing/preparation.

According to one embodiment, a machine learning model could be trainedto predict a characteristic about a WSI that is usually a proxy for theneed to do additional testing, e.g., presence of high-grade prostaticintraepithelial neoplasia (HGPIN) or ASAPs, etc. The output from thismodel could then be fed into a model to estimate the likelihood that aslide may need additional testing/preparation.

The above methods may be implemented using additional data regarding aspecific WSI. For example, according to one embodiment, additional datamay include one or more of (a) patient data such as genomic testing,family history, previous medical history, etc.; and/or (b) proceduredata such as physician notes/recommendation, observations from labtechnicians, etc.

Exemplary global outputs of the disclosed embodiments may containinformation or slide parameter(s) about an entire slide, e.g., thedepicted specimen type, the overall quality of the cut of the specimenof the slide, the overall quality of the glass pathology slide itself,or tissue morphology characteristics. Exemplary local outputs mayindicate information in specific regions of a slide, e.g., a particularslide region may be labeled as blurred or containing an irrelevantspecimen. The present disclosure includes embodiments for bothdeveloping and using the disclosed slide preparation automation, asdescribed in further detail below.

FIG. 1A illustrates a block diagram of a system and network fordetermining to order additional slides based on image analysis of tissuespecimens from digital pathology image(s), using machine learning,according to an exemplary embodiment of the present disclosure.

Specifically, FIG. 1A illustrates an electronic network 120 that may beconnected to servers at hospitals, laboratories, and/or doctors'offices, etc. For example, physician servers 121, hospital servers 122,clinical trial servers 123, research lab servers 124, and/or laboratoryinformation systems 125, etc., may each be connected to an electronicnetwork 120, such as the Internet, through one or more computers,servers, and/or handheld mobile devices. According to an exemplaryembodiment of the present application, the electronic network 120 mayalso be connected to server systems 110, which may include processingdevices that are configured to implement a disease detection platform100, which includes a slide analysis tool 101 for determining specimenproperty or image property information pertaining to digital pathologyimage(s), and using machine learning to determine to order an additionalslide, according to an exemplary embodiment of the present disclosure.

The physician servers 121, hospital servers 122, clinical trial servers123, research lab servers 124, and/or laboratory information systems 125may create or otherwise obtain images of one or more patients' cytologyspecimen(s), histopathology specimen(s), slide(s) of the cytologyspecimen(s), digitized images of the slide(s) of the histopathologyspecimen(s), or any combination thereof. The physician servers 121,hospital servers 122, clinical trial servers 123, research lab servers124, and/or laboratory information systems 125 may also obtain anycombination of patient-specific information, such as age, medicalhistory, cancer treatment history, family history, past biopsy orcytology information, etc. The physician servers 121, hospital servers122, clinical trial servers 123, research lab servers 124, and/orlaboratory information systems 125 may transmit digitized slide imagesand/or patient-specific information to server systems 110 over theelectronic network 120. Server system(s) 110 may include one or morestorage devices 109 for storing images and data received from at leastone of the physician servers 121, hospital servers 122, clinical trialservers 123, research lab servers 124, and/or laboratory informationsystems 125. Server systems 110 may also include processing devices forprocessing images and data stored in the storage devices 109. Serversystems 110 may further include one or more machine learning tool(s) orcapabilities. For example, the processing devices may include a machinelearning tool for a disease detection platform 100, according to oneembodiment. Alternatively or in addition, the present disclosure (orportions of the system and methods of the present disclosure) may beperformed on a local processing device (e.g., a laptop).

The physician servers 121, hospital servers 122, clinical trial servers123, research lab servers 124, and/or laboratory information systems 125refer to systems used by pathologists for reviewing the images of theslides. In hospital settings, tissue type information may be stored in aLIS 125.

FIG. 1B illustrates an exemplary block diagram of a disease detectionplatform 100 for determining to order additional slides based on imageanalysis of tissue specimens from digital pathology image(s), usingmachine learning, according to an exemplary embodiment of the presentdisclosure.

Specifically, FIG. 1B depicts components of the disease detectionplatform 100, according to one embodiment. For example, the diseasedetection platform 100 may include a slide analysis tool 101, a dataingestion tool 102, a slide intake tool 103, a slide scanner 104, aslide manager 105, a storage 106, and a viewing application tool 108.

The slide analysis tool 101, as described below, refers to a process andsystem for determining to order additional slides based on imageanalysis of tissue specimens from digital pathology image(s), usingmachine learning, according to an exemplary embodiment of the presentdisclosure.

The data ingestion tool 102 refers to a process and system forfacilitating a transfer of the digital pathology images to the varioustools, modules, components, and devices that are used for determining toorder additional slides based on image analysis of tissue specimens fromdigital pathology image(s), according to an exemplary embodiment.

The slide intake tool 103 refers to a process and system for scanningpathology images and converting them into a digital form, according toan exemplary embodiment. The slides may be scanned with slide scanner104, and the slide manager 105 may process the images on the slides intodigitized pathology images and store the digitized images in storage106.

The viewing application tool 108 refers to a process and system forproviding a user (e.g., pathologist) with specimen property or imageproperty information pertaining to digital pathology image(s), accordingto an exemplary embodiment. The information may be provided throughvarious output interfaces (e.g., a screen, a monitor, a storage device,and/or a web browser, etc.).

The slide analysis tool 101, and each of its components, may transmitand/or receive digitized slide images and/or patient information toserver systems 110, physician servers 121, hospital servers 122,clinical trial servers 123, research lab servers 124, and/or laboratoryinformation systems 125 over a network 120. Further, server systems 110may include storage devices for storing images and data received from atleast one of the slide analysis tool 101, the data ingestion tool 102,the slide intake tool 103, the slide scanner 104, the slide manager 105,and viewing application tool 108. Server systems 110 may also includeprocessing devices for processing images and data stored in the storagedevices. Server systems 110 may further include one or more machinelearning tool(s) or capabilities, e.g., due to the processing devices.Alternatively or in addition, the present disclosure (or portions of thesystem and methods of the present disclosure) may be performed on alocal processing device (e.g., a laptop).

Any of the above devices, tools, and modules may be located on a devicethat may be connected to an electronic network 120, such as the Internetor a cloud service provider, through one or more computers, servers,and/or handheld mobile devices.

FIG. 1C illustrates an exemplary block diagram of a slide analysis tool101, according to an exemplary embodiment of the present disclosure. Theslide analysis tool 101 may include a training image platform 131 and/ora target image platform 135.

According to one embodiment, the training image platform 131 may includea training image intake module 132, a stain module 133, and/or a recutmodule 134.

The training image platform 131, according to one embodiment, may createor receive training images that are used to train a machine learningmodel to effectively process, analyze, and classify digital pathologyimages. For example, the training images may be received from any one orany combination of the server systems 110, physician servers 121,hospital servers 122, clinical trial servers 123, research lab servers124, and/or laboratory information systems 125. Images used for trainingmay come from real sources (e.g., humans, animals, etc.) or may comefrom synthetic sources (e.g., graphics rendering engines, 3D models,etc.). Examples of digital pathology images may include (a) digitizedslides stained with a variety of stains, such as (but not limited to)H&E, Hematoxylin alone, IHC, molecular pathology, etc.; and/or (b)digitized tissue samples from a 3D imaging device, such as microCT.

The training image intake module 132 may create or receive a datasetcomprising one or more training images corresponding to either or bothof images of a human tissue and images that are graphically rendered.For example, the training images may be received from any one or anycombination of the server systems 110, physician servers 121, hospitalservers 122, clinical trial servers 123, research lab servers 124,and/or laboratory information systems 125. This dataset may be kept on adigital storage device. The stain module 133 may predict which newstains should be ordered for a selected slide due to a deficiency, basedon the received digital image(s) and received data. The recut module 134may predict whether a recut will be needed, based on the receiveddigital image(s) and received data.

According to one embodiment, the target image platform 135 may include atarget image intake module 136, a deficiency prediction module 137, andan output interface 138. The target image platform 135 may receive atarget image and apply the machine learning model to the received targetimage to determine to order an additional slide. For example, the targetimage may be received from any one or any combination of the serversystems 110, physician servers 121, hospital servers 122, clinical trialservers 123, research lab servers 124, and/or laboratory informationsystems 125. The target image intake module 136 may receive a targetimage corresponding to a target specimen. The deficiency predictionmodule 137 may apply the machine learning model to the target image tostain deficiencies and/or predict a needed recut associated with thetarget specimen.

The output interface 138 may be used to output information about thetarget image and the target specimen. (e.g., to a screen, monitor,storage device, web browser, etc.).

FIG. 2 is a flowchart illustrating an exemplary method of a tool fordetermining to order additional slides based on image analysis of tissuespecimens from digital pathology image(s), according to an exemplaryembodiment of the present disclosure. For example, an exemplary method200 (e.g., steps 202 to 206) may be performed by the slide analysis tool101 automatically or in response to a request from a user (e.g.,physician, pathologist, etc.).

According to one embodiment, the exemplary method 200 for determining toorder additional slides may include one or more of the following steps.In step 202, the method may include receiving a target image of a slidecorresponding to a target specimen, the target specimen comprising atissue sample from a patient. For example, the target image may bereceived from any one or any combination of the server systems 110,physician servers 121, hospital servers 122, clinical trial servers 123,research lab servers 124, and/or laboratory information systems 125.

In step 204, the method may include applying a machine learning model tothe target image to predict pathologist order information associatedwith the target specimen. The predicting the pathologist orderinformation may include determining a likelihood that the additionalslide is to be prepared based on specimen information of the targetspecimen, and determining, in response to the likelihood being greaterthan or equal than a predetermined amount, to automatically order theadditional slide to be prepared.

The machine learning model may be generated by processing a plurality oftraining images to predict stain order information and/or recut orderinformation, and the training images may include images of human tissueand/or images that are algorithmically generated. The machine learningmodel may be implemented using machine learning methods forclassification and regression. Training inputs could include real orsynthetic imagery. Training inputs may or may not be augmented (e.g.,adding noise). Exemplary machine learning models may include, but arenot limited to, any one or any combination of Neural Networks,Convolutional neural networks, Random Forest, Logistic Regression, andNearest Neighbor. Convolutional neural networks and other neural networkvariants may learn directly from pixels to learn features thatgeneralize well, but they typically require large amounts of trainingdata. The alternative exemplary models typically operate on featuresfrom a convolutional network or using hand-engineered computer visionfeature extraction techniques (e.g., SIFT, SURF, etc.), which often workless effectively if large amounts of data are available. The trainingimages may be received from any one or any combination of the serversystems 110, physician servers 121, hospital servers 122, clinical trialservers 123, research lab servers 124, and/or laboratory informationsystems 125. This dataset may be kept on a digital storage device.Images used for training may come from real sources (e.g., humans,animals, etc.) or may come from synthetic sources (e.g., graphicsrendering engines, 3D models, etc.). Examples of digital pathologyimages may include (a) digitized slides stained with a variety ofstains, such as (but not limited to) H&E, IHC, molecular pathology,etc.; and/or (b) digitized tissue samples from a 3D imaging device, suchas microCT.

In step 206, the method may include, based on the predicted pathologistorder information associated with the target specimen, determining toautomatically order an additional slide to be prepared. The additionalslide may be automatically ordered in response to the machine learningmodel identifying a diagnosis that automatically initiates an additionaltest. This diagnosis may be any one or any combination of lungadenocarcinoma, breast carcinoma, endometrioid adenocarcinoma, colonicadenocarcinoma, amyloid presence, and/or fungal organisms. Theadditional slide may be automatically ordered in response to the machinelearning model identifying a morphology that automatically triggers agenetic test. The morphology may be at least one of BAP1 deficient neviand/or succinate dehydrogenase deficient tumors. Ordering the additionalslide may include ordering a new stain to be prepared for the slidecorresponding to the target specimen and/or ordering a recut for theslide corresponding to the target specimen. The method may furtherinclude outputting an alert on a display indicating that the additionalslide is being prepared.

As illustrated in FIG. 3, according to one embodiment, exemplary methods300 and 320 for determining slide preparation parameter(s) may includeone or more of the steps below. In step 301, during a training phase,the method may include receiving a digital image of a pathology specimen(e.g., histology, cytology, etc.) in a digital storage device (e.g.,hard drive, network drive, cloud storage, RAM, etc.). The received imagemay be 2D (e.g., histology slides or unstained tissue cuts) or 3D (e.g.,micro CT, reconstructed 2D histology, etc.).

According to one embodiment, in step 303, the method may includereceiving an indication of whether a pathologist ordered new informationfor the specimen shown in the digital image. This step may includereceiving order information that a pathologist or other medicalprofessional associated with, or entered, for the specimen. New orderinformation might include additional stains, additional cuts, genomictesting, genetic testing, in-vitro lab tests, radiology imaging,computational (e.g., artificial intelligence) diagnostic tests, etc.

In step 305, the method may include training a machine learningalgorithm to predict whether and/or what order information may beassociated with one or more input/new digital images. This algorithm maybe implemented in multiple ways. For example, according to oneembodiment, the algorithm may be implemented by any one or anycombination of (1) machine learning algorithms and/or architectures,such as neural network methods, e.g., convolutional neural networks(CNNs) and recurrent neural networks (RNNs); (2) training methodologies,such as Multiple Instance Learning, Reinforcement Learning, ActiveLearning, etc.; (3) attribute/feature extraction including but notlimited to any one or any combination of estimated percentage of tissuein slide, base statistics on RGB, HSV or other color-space, and presenceof slide preparation issues or imaging artifacts such as bubbles, tissuefolds, abnormal staining, etc.; (4) using measure(s) of uncertainty inthe model predictions over other metrics as a proxy for needingadditional information; and (5) the output or associated metrics frommodels trained on a different task.

According to one or more embodiments, any of the above algorithms,architectures, methodologies, attributes, and/or features may becombined with any or all of the other algorithms, architectures,methodologies, attributes, and/or features. For example, any of themachine learning algorithms and/or architectures (e.g., neural networkmethods, convolutional neural networks (CNNs), recurrent neural networks(RNNs), etc.) may be trained with any of the training methodologies(e.g., Multiple Instance Learning, Reinforcement Learning, ActiveLearning, etc.)

The description of the terms below is merely exemplary and is notintended to limit the terms in any way.

A label may refer to information about an input to a machine learningalgorithm that the algorithm is attempting to predict.

For a given image of size N×M, a segmentation may be another image ofsize N×M that, for each pixel in an original image, assigns a numberthat describes the class or type of that pixel. For example, in a WSI,elements in the mask may categorize each pixel in the input image asbelonging to the classes of, e.g., background, tissue and/or unknown.

Slide level information may refer to information about a slide ingeneral, but not necessarily a specific location of that information inthe slide.

A heuristic may refer to a logic rule or function that deterministicallyproduces an output, given inputs. For example: if a prediction that aslide should be rescanned is greater than or equal to 32%, then outputone, if not, output 0. Another example heuristic may be that if beyond apredetermined percentage or portion of a slide is classified as unknown,then flag for re-scanning.

Embedding may refer to a conceptual high-dimensional numericalrepresentation of low-dimensional data. For example, if a WSI is passedthrough a CNN training to classify tissue type, the numbers on the lastlayer of the network may provide an array of numbers (e.g., in the orderof thousands) that contain information about the slide (e.g.,information about a type of tissue).

Slide level prediction may refer to a concrete prediction about a slideas a whole. For example, a slide level prediction may be that the slidehas a scanning issue, bubbles, tissue folds, etc. Further, slide levelprediction may refer to individual probability predictions over a set ofdefined classes (e.g., 33% chance of bubbles, 1% chance of tissue folds,99% chance of scanning artifacts, etc.).

A classifier may refer to a model that is trained to take input data andassociate it with a category.

According to one or more embodiments, the machine learning model may betrained in different ways. For example, the training of the machinelearning model may be performed by any one or any combination ofsupervised training, semi-supervised training, unsupervised trainingclassifier training, mixed training, and/or uncertainty estimation. Thetype of training used may depend on an amount of data, a type of data,and/or a quality of data. Table 1 below describes a non-limiting list ofsome types of training and the corresponding features.

TABLE 1 Index Input Label Model Output 1 WSI Segmentation CNN, RNN,Predicted Segmentation Embedding MLP Embedding 2 WSI Slide Level CNN,RNN, Embedding Embedding Information MLP Slide level prediction 3 WSI —CNN, RNN, Embedding Embedding MLP 4 Embedding Slide Level SVM, MLP,Slide level prediction Information RNN, Random Forests 5 Slide levelMeasure of MLP, RNN, Predict a likelihood that prediction how wrongStatistical an original prediction is the prediction Model wrong was

Supervised training may be used with a small amount of data to provide aseed for a machine learning model. In supervised training, the machinelearning model may look for a specific item (e.g., bubbles, tissuefolds, etc.), flag the slide, and quantify how much of the specific itemis present in the slide.

According to one embodiment, an example fully supervised training maytake as an input a WSI and may include a label of segmentation.Pipelines for a fully supervised training may include (1) 1; (2) 1,Heuristic; (3) 1, 4, Heuristic; (4) 1, 4, 5, Heuristic; and/or (5) 1, 5,Heuristic. Advantages of a fully supervised training may be that (1) itmay require fewer slides and/or (2) the output is explainable because(a) it may be known which areas of the image contributed to thediagnosis; and (b) it may be known why a slide is rejected (e.g.,bubbles found, tissue fold found, etc.). A disadvantage of using a fullysupervised training may be that it may require large amounts ofsegmentation which may be difficult to acquire.

According to one embodiment, an example semi-supervised (e.g., weaklysupervised) training may take as an input WSI and may include a label ofslide level information. Pipelines for a semi-supervised training mayinclude (1) 2; (2) 2, Heuristic; (3) 2, 4, Heuristic; (4) 2, 4, 5,Heuristic; and/or (5) 2, 5, Heuristic. Advantages of using asemi-supervised training may be that (1) the types of labels requiredmay be present in many hospital records; and (2) output is explainablebecause (a) it may be known which areas of the image contributed most tothe diagnosis; and (b) it may be known why a slide was rejected (e.g.,bubbles found, tissue fold found, etc.). A disadvantage of using asemi-supervised training is that it may be difficult to train. Forexample, the model may need to use a training scheme such as MultipleInstance Learning, Activate Learning, and/or distributed training toaccount for the fact that there is limited information about where inthe slide the information is that should lead to a decision.

According to one embodiment, an example unsupervised training may takeas an input a WSI and may require no label. The pipelines for anunsupervised training may include (1) 3, 4; and/or (2) 3, 4, Heuristic.An advantage of unsupervised training may be that it does not requireany labels. Disadvantages of using an unsupervised training may be that(1) it may be difficult to train. For example, it may need to use atraining scheme such as Multiple Instance Learning, Activate Learning,and/or distributed training to account for the fact that there islimited information about where in the slide the information is thatshould lead to a decision; (2) it may require additional slides; and/or(3) it may be less explainable because it might output a prediction andprobability without explaining why that prediction was made.

According to one embodiment, an example mixed training may includetraining any of the example pipelines described above for fullysupervised training, semi-supervised training, and/or unsupervisedtraining, and then use the resulting model as an initial point for anyof the training methods. Advantages of mixed training may be that (1) itmay require less data; (2) it may have improved performance; and/or (3)it may allow a mixture of different levels of labels (e.g.,segmentation, slide level information, no information). Disadvantages ofmixed training may be that (1) it may be more complicated and/orexpensive to train; and/or (2) it may require more code that mayincrease a number and complexity of potential bugs.

According to one embodiment, an example uncertainty estimation mayinclude training any of the example pipelines described above for fullysupervised training, semi-supervised training, and/or unsupervisedtraining, for any task related to slide data using uncertaintyestimation in the end of the pipeline. Further, a heuristic orclassifier may be used to predict whether a recut should be performedbased on an amount of uncertainty in the prediction of the test. Anadvantage of uncertainty estimation may be that it is robust toout-of-distribution data. For example, when unfamiliar data ispresented, it may still correctly predict that it is uncertain.Disadvantages of uncertainty estimation may be that (1) it may need moredata; (2) it may have poor overall performance; and/or (3) it may beless explainable because the model might not necessarily identify how aslide or slide embedding is abnormal.

According to one embodiment, an ensembles training may includesimultaneously running models produced by any of the example pipelinesdescribed above, and combining the outputs by a heuristic or aclassifier to produce robust and accurate results. Advantages ofensembles training may be that (1) it is robust to out-of-distributiondata; and/or (2) it may combine advantages and disadvantages of othermodels, resulting in a minimization of disadvantages (e.g., a supervisedtraining model combined with an uncertainty estimation model, and aheuristic that uses a supervised model when incoming data is indistribution and uses an uncertainty model when data is out ofdistribution, etc.). Disadvantages of ensembles training may be that (1)it may be more complex; and/or (2) it may be expensive to train and run.

Training techniques discussed herein may also proceed in stages, whereimages with greater annotations are initially used for training, whichmay allow for more effective later training using slides that have fewerannotations, are less supervised, etc.

Training may begin using the slides that are the most thoroughlyannotated, relative to all the training slide images that may be used.For example, training may begin using supervised learning. A first setof slides images may be received or determined with associatedannotations. Each slide may have marked and/or masked regions and mayinclude information such as whether the slide should be rejected. Thefirst set of slides may be provided to a training algorithm, for examplea CNN, which may determine correlations between the first set of slidesand their associated annotations.

After training with the first set of images is completed, a second setof slide images may be received or determined having fewer annotationsthan the first set, for example with partial annotations. In oneembodiment, the annotations might only indicate that the slide has adiagnosis or quality issue associated with it, but might not specifywhat or where disease may be found, etc. The second set of slide imagesmay be trained using a different training algorithm than the first, forexample Multiple Instance Learning. The first set of training data maybe used to partially train the system, and may make the second traininground more effective at producing an accurate algorithm.

In this way, training may proceed in any number of stages, using anynumber of algorithms, based on the quality and types of the trainingslide images. These techniques may be utilized in a situations wheremultiple training sets of images are received, which may be of varyingquality, annotation levels, and/or annotation types.

According to one embodiment, an exemplary method 320 for using the toolmay include one or more of the steps below. In step 321, the method mayinclude receiving a digital image of a pathology specimen (e.g.,histology, cytology, etc.) in a digital storage device (e.g., harddrive, network drive, cloud storage, RAM, etc.). In step 323, the methodmay include applying the algorithm from the training procedure (e.g.,method 300) to predict the likelihood that the digital image providesinsufficient information for a diagnosis and/or the likelihood that thedigital image may be associated with order information for an improvedpathology slide to be made. This prediction may be performed at thelevel of a specimen, a slide, a tissue block, etc.

In step 325, the method may include predicting order informationassociated with the received digital image. The order information mayinclude a type of testing or slide parameter(s) to order. Testing mayinclude additional stains, additional cuts, genomic testing, genetictesting, in-vitro lab tests, radiology imaging, computational (e.g.,artificial intelligence) diagnostic tests, etc. This prediction may beoutput to an electronic storage device. In one embodiment, the predictedlikelihood may be used to automatically trigger an order for newinformation from a histology technician. The predicted likelihood orpredicted order information may be used to prompt an automaticpreparation pipeline to prepare one or more additional slides (e.g.,re-cuts, staining, etc.). The automatic trigger or slide preparation maybe performed by a heuristic or auxiliary system. Alternately or inaddition, step 325 may include generating a visual indicator to alert auser (e.g., a pathologist, histology technician, etc.) that newinformation on a slide may be desired to make a diagnosis. A user maythen order new information, based on the alert. The alert may allow auser to initiate preparation for a new slide earlier, rather than later.

Techniques discussed herein provide a heuristic for determining whetherto produce a stain, and a method for streamlining pathology slideanalysis. One aspect of streamlining slide analysis includes orderingnew slide(s) and/or automating slide order information. The slide ordermachine learning embodiments described above present solution(s) forthis aspect. Another aspect of streamlining pathology slide analysis mayinclude minimizing an expected cost of running additionaltests/generating additional specimen slides. According to oneembodiment, minimizing the cost may follow the function below:min((A+B)*FN(th)+C*FP(th)), where

A: Average cost of a delayed diagnosis;

B: Average cost for a pathologist to decide additional staining isrequired;

C: Cost of additional test;

FN(th): False negative rate across the validation set as a function ofthreshold; and

FP(th): False positive rate in across the validation set as a functionof threshold.

The above-described training and usage phases may include embodimentsusable in research and/or production/clinical/industrial settings. Theseare described in detail below.

According to one embodiment, a method may include predicting when a newstain is ordered by a pathologist. For example, when a pathologist isstruggling with a diagnosis or finds specific borderline signs ofcancer, the pathologist may request for a slide to be prepared with anadditional stain, e.g., immunohistochemistry (IHC), molecular pathology,Congo Red, etc.

According to one embodiment illustrated in FIG. 4, an exemplary method400 for developing a stain order prediction tool may include one or moreof the steps below. In step 401, the method may include receiving afirst digital image of a first slide comprising a pathology specimen(e.g., histology, cytology, etc.) in a digital storage device (e.g.,hard drive, network drive, cloud storage, RAM, etc.). In step 403, themethod may include receiving, for the first slide, an indication ofwhether a pathologist ordered a new stain for that slide. This step mayinclude receiving a stain order associated with the first slide. Forexample, the indication for the slide may state the exact stain that wasordered. Additional information about the specimen of the slide may alsobe received, e.g., data on the tissue type from with the specimen wastaken and/or any diagnostic data associated with the patient or caseassociated with the specimen.

In step 405, the method may include training a machine learningalgorithm to receive a second digital image of a pathology specimen andreceive data (e.g., slide order information) associated with the seconddigital image. A trained machine learning algorithm may then predictwhether a new stain was ordered for a selected slide, based on thereceived digital image(s) and received data (e.g., step 405). Thetrained machine learning algorithm may also predict which (new) stainswere ordered for a selected slide, based on the received digitalimage(s) and received data (e.g., step 405). This algorithm may beimplemented in multiple ways by using any combination of (1) Neuralnetworks such as CNNs, RNNs, etc.; (2) Training methodologies, such asMultiple Instance Learning, Reinforcement Learning, Active Learning,etc.; (3) Feature extraction including but not limited to any one or anycombination of percentage of tissue in slide, base statistics on RGB,HSV or other color-spaces, a presence of slide preparation or imagingartifacts such as bubbles, tissue folds, abnormal staining, etc.; and(4) simple classification methods, such as random forest, support vectormachine (SVM), multiplayer perceptron (MLP), etc. The above descriptionof machine learning algorithms for FIG. 3 (e.g., Table 1 andcorresponding description) may also apply to the machine learningalgorithms of FIG. 4.

An exemplary method 420 for using the disclosed stain order predictiontool may include one or more of the steps below. In step 421, the methodmay include receiving one or more digital images of a slide of apathology specimen (e.g., histology, cytology, etc.) in a digitalstorage device (e.g., hard drive, network drive, cloud storage, RAM,etc.) (e.g., step 421). Information about the specimen may be received,e.g., a tissue type from which the specimen harvested and/or anydiagnostic data associated with the selected patient or selected case.In step 423, the method may include predicting, using the trainedmachine learning algorithm (e.g., of method 400) the likelihood that anew stain is desired for the slide. Step 423 may also include predictinga stain order for the slide.

In step 425, the method may include outputting the prediction to anelectronic storage device. The predicted likelihood or predicted stainorder may be used to automatically trigger an order for a histologytechnician. In one embodiment, a visual indicator may be generated toalert a user (e.g., a pathologist, histology technician, etc.) that anew stain may be desired, so that the user may promptly order the newstain. Alternately, or in addition, the predicted likelihood orpredicted stain order may be used as part of an automated slide stainingpipeline to prepare one or more slides with the required stain. Examplemethods include, but are not limited to, low model information,predicting high risk lesions, identifying diagnoses that mayautomatically need additional tests, and identifying suspiciousmorphology that automatically triggers genetic testing.

Examples of diagnoses that may automatically need additional tests mayinclude any one or any combination of (1) Lung adenocarcinoma triggers apanel of immunostains and recuts for molecular testing (e.g., EGFR(Epidermal Growth Factor Receptor), KRAS (Kirsten RAt Sarcoma), ALK(anaplastic lymphoma receptor tyrosine kinase), ROS, BRAF (B-Rafproto-oncogene), MET (MET Proto-Oncogene, Receptor Tyrosine Kinase),etc.); (2) Breast carcinoma triggers a hormone receptor immunostainpanel (e.g., ER (oestrogen receptor), PR (progesterone receptor)<Her2(human epidermal growth factor receptor type 2)); (3) Endometrioidadenocarcinoma and colonic adenocarcinoma trigger mismatch repairimmunostains (e.g., MLH1, MSH2, PMS2, MSH6 genes; (4) Amyloid presencetriggers Congo Red; and (5) Fungal organisms trigger, e.g., PAS(Periodic acid-Schiff) and GMS (Grocott methenamine silver).

Examples of suspicious morphology that automatically trigger genetictesting may include (1) BAP1 deficient nevi, triggers BAP1 immunostain;and/or (2) succinate dehydrogenase deficient tumors triggers SDH(succinate dehydrogenase) immunostain.

According to one embodiment, some diagnoses and/or stain orderpredictions may prompt at least one additional stain that may betriggered automatically, e.g., if the algorithm of method 420 hasdetermined a diagnosis within a threshold or certainty and/or determinedone set of stain order information. Additionally, some features of thepathology images may be subtle and additional stains may assist thepathologist to determine a diagnosis. In one embodiment, the additionalstain(s) may be prompted/ordered once the algorithm of method 420detects that an image enhancement or improved slide is desired.

According to one embodiment, examples of situations in which at leastone additional stain may be triggered automatically may includediagnoses that trigger one or more immunostains. For example, lungadenocarcinoma may trigger a panel of immunostains and recuts formolecular testing (EGFR, KRAS, ALK, ROS, BRAF, MET, etc.). Additionally,breast carcinoma may trigger a hormone receptor immunostain panel (ER,PR<Her2). Also, endometrioid adenocarcinoma and colonic adenocarcinomamay trigger mismatch repair immunostains (MLH1, MSH2, PMS2, MSH6).

According to one embodiment, a pathology image may include certainfeatures that are subtle and difficult to detect. In this case, anautomatic ordering of more stains may be triggered to enhance somefeatures to assist a pathologist in determining a diagnosis. Forexample, a BAP1 deficient nevi detected by the algorithm may predicttumor predisposition, and a BAP1 immunostain may be ordered. As anotherexample, if a succinate dehydrogenase deficient tumor is recognized, anSDH immunostain may be ordered. As another example, if amyloid isdetected, a Congo red stain may be ordered to highlight the amyloid. Asanother example, if fungal organisms are detected by the algorithm, aPeriodic acid-Schiff (PAS) and/or Gomori's methenamine silver (GMS)stain may be ordered to highlight the fungal organisms.

According to one embodiment, a method may include predicting when arecut is to be ordered by a pathologist. For example, when a pathologistdetects a possible border of cancer in a slide, or when a pathologistdetects that a slide does not capture enough of a specimen's crosssection to render a diagnosis, the pathologist may request for anadditional cut to be made from the specimen, and a new slide to beprepared.

According to one embodiment illustrated in FIG. 5, an exemplary method500 for developing a recut order prediction tool may include one or moreof the steps below. In step 501, the method may include receiving adigital image of a first slide comprising a pathology specimen (e.g.,histology, cytology, etc.) in a digital storage device (e.g., harddrive, network drive, cloud storage, RAM, etc.). In step 503, the methodmay include receiving an indication for a tissue block of the specimen,of whether the pathologist ordered a recut for that tissue block. Thisstep may include receiving a recut location of the tissue block,associated with the first slide. For example, the indication for eachblock could state exactly where the recut was ordered (e.g., above orbelow each slide in the block). Additional information about thespecimen may also be received, e.g., data on the tissue type from withthe specimen was taken and/or any diagnostic data associated with thepatient or case associated with the specimen. Other examples ofadditional information may include information about the grossdescription of the specimen (e.g., images of the gross specimen, testdescription, size and shape dimensions, etc.).

According to one embodiment, in step 505, the method may includetraining a machine learning algorithm to predict whether a recut wasordered for an input slide, based on received digital images ofpathology specimen(s) and additional information corresponding to eachdigital image/pathology specimen. For example, the resultant trainedmachine learning algorithm may predict whether a recut was ordered foreach tissue block and/or predict a location of the recut (e.g., above orbelow the cut of an input slide associated with the tissue block). Thisalgorithm could be implemented in multiple ways by using any combinationof (1) neural networks such as CNNs, recurrent neural networks (RNNs),etc.; (2) training methodologies such as Multiple Instance Learning,Reinforcement Learning, Active Learning, etc.; (3) feature extractionincluding but not limited to (a) percentage of tissue in slide, (b) basestatistics on RGB (red, green, blue), HSV (hue, saturation, value), HSL(hue, saturation, lightness), or other color-spaces, and (c) a presenceof slide preparation or imaging artifacts such as bubbles, tissue folds,abnormal staining, etc.; and/or (4) simple classification methods suchas random forest, SVM, MLP, etc.

An exemplary method 520 for developing a recut order prediction tool mayinclude one or more of the steps below. In step 521, the method mayinclude receiving one or more digital images of a slide of a pathologyspecimen (e.g., histology, cytology, etc.) in a digital storage device(e.g., hard drive, network drive, cloud storage, RAM, etc.). Informationabout the specimen may be received, e.g., a tissue type from which thespecimen harvested and/or any diagnostic data associated with theselected patient or selected case. In step 523, the method may includepredicting, using the trained machine learning algorithm (e.g., ofmethod 500), the likelihood that a recut is desired for a tissue blockassociated with the specimen. Step 523 may also include predicting arecut order (e.g., recut location) for the slide.

According to one embodiment, in step 525, the method may includeoutputting the prediction to an electronic storage device (e.g., step525). The predicted likelihood or predicted recut order may be used toautomatically trigger an order for a histology technician. In oneembodiment, a visual indicator may be generated to alert a user (e.g., apathologist, histology technician, etc.) that a new stain may bedesired, so that the user may promptly order the new stain. In oneembodiment, an output may include prompting an automatic slidesegmenting machine to cut one or more additional slides from the tissueblock associated with the specimen. An output may further include adetermination of the recut location (e.g., how deep into the tissue tocut) and/or the axis for the next recut order. In one embodiment, anadditional system may be used to compute precise parameters forgenerating the recut (e.g., recut location, axis, etc.). Some examplemethods for determining or computing recut order information mayinclude, but are not limited to (1) from the past N cuts, estimate theamount of tissue to be present in a slide as a function of the locationfrom where a prior specimen was cut and maximize said function topredict the next best location to cut; (2) if small/ambiguous signs ofpathogens or cancer are detected, order a recut close (e.g., within apredetermined distance/distance threshold) to the first location/depthto increase the amount of information collected about that suspiciousregion until ambiguity is resolved; and/or (3) if grading is ambiguous,order a recut close (e.g., within a predetermined distance/distancethreshold) to the first location/depth to increase the amount ofinformation collected about that suspicious region until ambiguity isresolved.

As shown in FIG. 6, device 600 may include a central processing unit(CPU) 620. CPU 620 may be any type of processor device including, forexample, any type of special purpose or a general-purpose microprocessordevice. As will be appreciated by persons skilled in the relevant art,CPU 620 also may be a single processor in a multi-core/multiprocessorsystem, such system operating alone, or in a cluster of computingdevices operating in a cluster or server farm. CPU 620 may be connectedto a data communication infrastructure 610, for example, a bus, messagequeue, network, or multi-core message-passing scheme.

Device 600 also may include a main memory 640, for example, randomaccess memory (RAM), and also may include a secondary memory 630.Secondary memory 630, e.g., a read-only memory (ROM), may be, forexample, a hard disk drive or a removable storage drive. Such aremovable storage drive may comprise, for example, a floppy disk drive,a magnetic tape drive, an optical disk drive, a flash memory, or thelike. The removable storage drive in this example reads from and/orwrites to a removable storage unit in a well-known manner. The removablestorage unit may comprise a floppy disk, magnetic tape, optical disk,etc., which is read by and written to by the removable storage drive. Aswill be appreciated by persons skilled in the relevant art, such aremovable storage unit generally includes a computer usable storagemedium having stored therein computer software and/or data.

In alternative implementations, secondary memory 630 may include othersimilar means for allowing computer programs or other instructions to beloaded into device 600. Examples of such means may include a programcartridge and cartridge interface (such as that found in video gamedevices), a removable memory chip (such as an EPROM, or PROM) andassociated socket, and other removable storage units and interfaces,which allow software and data to be transferred from a removable storageunit to device 600.

Device 600 also may include a communications interface (“COM”) 660.Communications interface 660 allows software and data to be transferredbetween device 600 and external devices. Communications interface 660may include a modem, a network interface (such as an Ethernet card), acommunications port, a PCMCIA slot and card, or the like. Software anddata transferred via communications interface 660 may be in the form ofsignals, which may be electronic, electromagnetic, optical, or othersignals capable of being received by communications interface 660. Thesesignals may be provided to communications interface 660 via acommunications path of device 600, which may be implemented using, forexample, wire or cable, fiber optics, a phone line, a cellular phonelink, an RF link or other communications channels.

Device 600 also may include input and output ports 650 to connect withinput and output devices such as keyboards, mice, touchscreens,monitors, displays, etc. Of course, the various server functions may beimplemented in a distributed fashion on a number of similar platforms,to distribute the processing load. Alternatively, the servers may beimplemented by appropriate programming of one computer hardwareplatform.

Throughout this disclosure, references to components or modulesgenerally refer to items that logically can be grouped together toperform a function or group of related functions. Like referencenumerals are generally intended to refer to the same or similarcomponents. Components and modules can be implemented in software,hardware, or a combination of software and hardware.

The tools, modules, and functions described above may be performed byone or more processors. “Storage” type media may include any or all ofthe tangible memory of the computers, processors or the like, orassociated modules thereof, such as various semiconductor memories, tapedrives, disk drives and the like, which may provide non-transitorystorage at any time for software programming.

Software may be communicated through the Internet, a cloud serviceprovider, or other telecommunication networks. For example,communications may enable loading software from one computer orprocessor into another. As used herein, unless restricted tonon-transitory, tangible “storage” media, terms such as computer ormachine “readable medium” refer to any medium that participates inproviding instructions to a processor for execution.

The foregoing general description is exemplary and explanatory only, andnot restrictive of the disclosure. Other embodiments of the inventionwill be apparent to those skilled in the art from consideration of thespecification and practice of the invention disclosed herein. It isintended that the specification and examples be considered as exemplaryonly.

What is claimed is:
 1. A computer-implemented method for processing anelectronic image corresponding to a specimen, the method comprising:receiving a target electronic image of a slide corresponding to a targetspecimen, the target specimen comprising a tissue sample from a patient;applying a machine learning system to the target electronic image todetermine deficiencies associated with the target specimen, the machinelearning system having been generated by processing a plurality oftraining images to predict stain deficiencies, the training imagescomprising either or both of images of human tissue and images that arealgorithmically generated; and based on the deficiencies associated withthe target specimen, determining to automatically order an additionalslide to be prepared using immunohistochemistry.
 2. Thecomputer-implemented method of claim 1, wherein determining deficienciescomprises determining a likelihood that the additional slide is to beprepared based on specimen information of the target specimen, and inresponse to the likelihood being greater than or equal to apredetermined amount, automatically ordering the additional slide to beprepared.
 3. The computer-implemented method of claim 1, wherein theadditional slide is automatically ordered in response to the machinelearning system identifying a diagnosis that automatically initiates anadditional test.
 4. The computer-implemented method of claim 1, whereinthe additional slide is automatically ordered in response to the machinelearning system identifying a diagnosis that automatically initiates anadditional test, and wherein the diagnosis that automatically initiatesthe additional test comprises additional test is automatically initiatedin response to a diagnosis of any one or any combination of lungadenocarcinoma, breast carcinoma, endometrioid adenocarcinoma, colonicadenocarcinoma, adenocarcinoma in other tissues, sarcomas, prognosticbiomarkers, suspicious lesions, amyloid presence, and/or fungalorganisms.
 5. The computer-implemented method of claim 1, wherein theadditional slide is automatically ordered in response to the machinelearning system identifying a morphology that automatically triggers agenetic test.
 6. The computer-implemented method of claim 1, wherein theordering the additional slide comprises ordering a new stain to beprepared for the slide corresponding to the target specimen.
 7. Thecomputer-implemented method of claim 1, wherein the ordering theadditional slide comprises ordering a recut for the slide correspondingto the target specimen.
 8. The computer-implemented method of claim 1,further comprising outputting an alert on a display indicating that theadditional slide is being prepared.
 9. A system for processing anelectronic image corresponding to a specimen, the system comprising: atleast one memory storing instructions; and at least one processorconfigured to execute the instructions to perform operations comprising:receiving a target electronic image of a slide corresponding to a targetspecimen, the target specimen comprising a tissue sample from a patient;applying a machine learning system to the target electronic image todetermine deficiencies associated with the target specimen, the machinelearning system having been generated by processing a plurality oftraining images to predict stain deficiencies, the training imagescomprising either or both of images of human tissue and images that arealgorithmically generated; and based on the deficiencies associated withthe target specimen, determining to automatically order an additionalslide to be prepared using immunohistochemistry.
 10. The system of claim9, wherein determining deficiencies comprises determining a likelihoodthat the additional slide is to be prepared based on specimeninformation of the target specimen, and in response to the likelihoodbeing greater than or equal to a predetermined amount, determining toautomatically order the additional slide to be prepared.
 11. The systemof claim 9, wherein the additional slide is automatically ordered inresponse to the machine learning system identifying a diagnosis thatautomatically initiates an additional test.
 12. The system of claim 9,wherein the additional slide is automatically ordered in response to themachine learning system identifying a diagnosis that automaticallyinitiates an additional test, and wherein the diagnosis thatautomatically initiates the additional test comprises any one or anycombination of lung adenocarcinoma, breast carcinoma, endometrioidadenocarcinoma, colonic adenocarcinoma, adenocarcinoma in other tissues,sarcomas, prognostic biomarkers, suspicious lesions, amyloid presence,and/or fungal organisms.
 13. The system of claim 9, wherein theadditional slide is automatically ordered in response to the machinelearning system identifying a morphology that automatically triggers agenetic test.
 14. The system of claim 9, wherein the ordering theadditional slide comprises ordering a new stain to be prepared for theslide corresponding to the target specimen.
 15. The system of claim 9,wherein the ordering the additional slide comprises ordering a recut forthe slide corresponding to the target specimen.
 16. The system of claim9, further comprising outputting an alert on a display indicating thatthe additional slide is being prepared.
 17. A non-transitorycomputer-readable medium storing instructions that, when executed byprocessor, cause the processor to perform a method for processing anelectronic image corresponding to a specimen, the method comprising:receiving a target electronic image of a slide corresponding to a targetspecimen, the target specimen comprising a tissue sample from a patient;applying a machine learning system to the target electronic image todetermine deficiencies associated with the target specimen, the machinelearning system having been generated by processing a plurality oftraining images to predict stain deficiencies, the training imagescomprising either or both of images of human tissue and images that arealgorithmically generated; and based on the deficiencies associated withthe target specimen, determining to automatically order an additionalslide to be prepared using immunohistochemistry.
 18. The non-transitorycomputer-readable medium of claim 17, wherein determining deficienciescomprises determining a likelihood that the additional slide is to beprepared based on specimen information of the target specimen, and inresponse to the likelihood being greater than or equal to apredetermined amount, determining to automatically order the additionalslide to be prepared.
 19. The non-transitory computer-readable medium ofclaim 17, wherein the additional slide is automatically ordered inresponse to the machine learning system identifying a diagnosis thatautomatically initiates an additional test.
 20. The non-transitorycomputer-readable medium of claim 17, wherein the additional slide isautomatically ordered in response to the machine learning systemidentifying a diagnosis that automatically initiates an additional test,and wherein the diagnosis that automatically initiates the additionaltest comprises any one or any combination of lung adenocarcinoma, breastcarcinoma, endometrioid adenocarcinoma, colonic adenocarcinoma,adenocarcinoma in other tissues, sarcomas, prognostic biomarkers,suspicious lesions, amyloid presence, and/or fungal organisms.